Repository navigation
feat(knowledge): unify search across cases, skills, and records - #466
jiaran-king wants to merge 10 commits into
Conversation
…PyYAML Each casebook entry already recorded the directions tried, their outcome and when a retry would be justified, but only as prose or a Markdown table that ranking could not weigh. Add a `findings` list to all seven cases: direction, outcome, recorded reason, revisit condition, retrieval keywords and `scope`. Every text field is taken from the existing case body; no claim, cause or measurement is added and no existing field changes. `scope` states what would have to change for an observation to stop holding: `local` (that host or harness), `regime_bound` (the cohort the case names) or `general` (a property of the method). It is not derivable from the outcome -- a rejection can be bound to one resolution or hold everywhere. Of 35 findings, 24 are regime_bound, 6 general and 5 local. The source-contract cases are regime_bound: each reads one pinned upstream commit and says so in its own applicability, and a later revision can change the dispatch it describes. Reading a case parses no skill frontmatter, but importing the skill catalog required PyYAML, so a base install could not discover or open one. The parser is now imported where frontmatter is parsed. Its absence raises MissingSkillsExtra, a ValueError subclass in a module of its own so catching it never imports the parser, and `collect` lets it through rather than reporting "nothing found" for an install problem. The snapshot tool relied on the old import-time failure to stop before side effects, and now asks for the parser explicitly at that point. The reader exposes `cases()` and `case_resource_path()` so a caller ranking cases resolves them by the same rule `read` uses. Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=max] (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
…ndings Mechanical follow-up. The generator pins each card's source hashes to a revision, so the three skills that own the edited cases -- the optimization casebook, the kernel-writing guide and the search policy -- held hashes for the case files as they were before findings were added, and `indexed_skill` rejected those projections as stale. Regenerated with tools/build_skill_index.py against the previous commit. Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=max] (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Port the verified find and read flow to the current Knowledge package and Apps-facing CLI. Keep query fixtures, compact exact reads, and standalone plugin coverage together. Signed-off-by: Zhou ziheng <jiaranran2@gmail.com> Agent-assisted-by: Claude Code (author) Agent-assisted-by: Codex (author)
Skill evaluationPLANNED: not executed; there are no task outcomes yet.
Author verification steps (nonpaid; not yet run)
python tools/build_skill_index.py --source-revision 8288a004c333b0e338b3ad31fe823052a142b90a --checkThe declared source pin matches this head's canonical library bytes. The command has not run: retain its exit code and full drift/success output. It checks the generated projection, not claim truth or skill efficacy.
python -m json.tool optimization-search-policy/plan/tasks.json
python -m json.tool optimization-search-policy/plan/schema.jsonUncovered scope by location: Canonical library paths: 6; Generated projection paths: 4; Other paths: 22. Path counts are not claim counts; no metadata-only conclusion is inferred. Evidence and logs · Retention: 30 days. Plan bindings, budget, frozen exam and advisory detailsAdvisory: obtain missing evidenceRetained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Model/settings:
Frozen exam hashes: Selection/task metadata (not a relevance verdict): {"adapter":"kv-admission-diagnostic-v1","basis":"trusted_path_mapping","matched_paths":["src/inferrsi/knowledge/data/skill_library/methods/optimization-search-policy/cases/sarathi-chunk-budget.json"],"required_answer_fields":["diagnosis","evidence","next_check","reuse_authorized"],"tasks":[{"id":"case-05","provenance":"Synthetic finite-state observations; not a production trace or engine benchmark.","request":"Diagnose this blocked request at the recorded decision boundary; propose the next read-only check.","split":"holdout"}]}Automatic execution / cumulative PR budget: {"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}Projection check selection (not an executed result): {"declared_in":"src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json","source_revision":"8288a004c333b0e338b3ad31fe823052a142b90a","status":"ready"}Provider/settings: Uncovered / unsupported paths (32)
|
Agent-assisted-by: Codex (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
|
Posted by an agent on behalf of @Dong1017 (incremental review only; first review on this PR, no earlier human findings to repeat). PR review verdict — DO NOT PASS Reviewed base The Blockers
Optional (not blocking)
Not verified: the ranking-quality table in the PR body (16/17, 21/34, 7/9, 9/9) and the 210,721 → 10,950 byte read-back reduction; I did not rerun the labelled-query fixtures against #380. Checking scope: read the PR body, #379, the CI check list and the failed |
|
审阅了本 PR 的 base
|
david6666666
left a comment
There was a problem hiding this comment.
One compatibility issue with the documented discovery path.
| entries. Add `--kinds case` when you specifically need a past observation. | ||
| Open a promising hit using its displayed `open` command or the JSON | ||
| `open_argv` argument list. If the result says `cannot open`, check the reason | ||
| and consult another hit when appropriate. |
There was a problem hiding this comment.
Could we keep the packaged discovery fallback here? On a host without the optional PyYAML extra, find 'ascend graph mode' --kinds skill ranks the skill but offers no open_argv: skill hits use checkout: and packaged projections are filtered out. Trying another skill hit won't help, while collect --relevance 'ascend graph mode' --limit 1 --material --json still returns the packaged skill on the same host. Please point this case to collect, or have find offer the corresponding packaged read command.
Use the existing case bodies to express the recorded decisions, limits, and follow-up checks. Keep the tests focused on which outcome is retrieved and the evidence it carries. Agent-assisted-by: Codex / gpt-6.1-sol [effort=max] (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Agent-assisted-by: Codex / gpt-6.1-sol [effort=max] (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
|
审阅了自上次评论以来的增量: 上次的两点阻塞都还在:
另有两点请在这个 head 上处理或更新:
|
Use the packaged projection and its revision to resolve a Skill that the base installation cannot open from find. Agent-assisted-by: Codex (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base 016c7c7df023944f0f46d8e777d6d31f1df1764c and head 866fd2aa2816652976c107d6ab09eb80d0343ef1.
Re-review after docs push that adds the packaged-skill PyYAML fallback guidance. Independent of author jiaran-king.
Blockers
-
knowledge-model-policystill noncompliant on this head.declarationsfailed withMaterial AI author 1: model is unapproved or suspended(run 43846776491). PR-body author 1 still hasmodel_id/effort=unknown. Keep unknown values truthful; obtain an approval-owner exception perdocs/knowledge-model-policy.md, or revise protected-path authorship to qualifying declarations. Do not invent model ids. -
Default discovery promotion exceeds accepted #379 scope. #379 is still
status:proposedand keeps the product decision: do not promote the #380 short guidance to the product entry; temporarily keep the existing catalogue andrg. This head still makesfind → readthe plugin/README/quickstart default (plugins/inferrsi/skills/inferrsi/SKILL.md,README.md,docs/quickstart.md). Expected: requester/maintainer acceptance on #379 forfindas default, or narrow this PR to additivefind/read --omit-resourceswithout replacing the default discovery path. -
Branch is CONFLICTING vs current
main.mergeable=false,mergeable_state=dirty. Conflicting paths include at leastplugins/inferrsi/skills/inferrsi/scripts/inferrsi.py,tests/check_installed.py, andtests/test_knowledge_cli.py. Rebase/resolve onto currentmainbefore any Approve/merge.
Not blocking / observed
- David's packaged fallback note (discussion_r4583888248) looks addressed in the plugin skill text on this head (
collect --relevance→resourcewhen PyYAML is missing). Keep that path through rebase. - Ranking implementation direction remains sound; ranking-quality table numbers were not re-run here.
- No merge while dirty or while blockers 1–2 remain.
Checked / not checked
Checked: PR body, #379 accepted wording, declarations failure log, conflict paths via merge-tree, plugin SKILL fallback section, prior Dong/Amy/David findings. Did not re-run the labelled-query suites or full pytest on this conflicting head.
david6666666
left a comment
There was a problem hiding this comment.
One follow-up on the documented PyYAML fallback.
| `open_argv` argument list. | ||
|
|
||
| When a Skill hit is unreadable because PyYAML is missing, use | ||
| `collect --relevance "<same query>" --limit 1 --material --json` to look up |
There was a problem hiding this comment.
Could the fallback verify that it found the same skill? Without PyYAML, find 'cuda graph warmup' --kinds skill --limit 1 selects kernel-writing-guide, but the documented collect --relevance command selects ascend-aclgraph-capture-replay. Please match skill_documents.original to the selected hit's path, or describe this as a fresh search rather than opening the selected hit.
|
审阅了自上次评论以来的增量: 上次的两点阻塞都还在:
关于上次第 3 点:本次文档补充了 上次第 4 点仍然存在:正文「实测结果」第三行仍写在 当前 head 上 cursor[bot] 的 review(5918404550)为 CHANGES_REQUESTED / DO NOT PASS,其中知识模型策略与 #379 范围两点与上述 1、2 一致。本次增量是文档,不涉及新的精度或性能对比实验。我没有重跑检索对照,也没有在本机验证无 PyYAML 回退命令。 |
jiangkuaixue123
left a comment
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed target main, PR base 016c7c7df023944f0f46d8e777d6d31f1df1764c and head 866fd2aa2816652976c107d6ab09eb80d0343ef1. Also checked integration against fetched main at 6c7e237935a22cc5e90e954d4f397376e753e7d8.
Delegation: the user directly requested an independent review of #466 and authorized posting review findings. Acting reviewer: Codex under @jiangkuaixue123 (repository write access verified); scope: code, tests and merge readiness; permitted action: publish this revision-bound Request changes review. This reviewer did not author or materially contribute to this PR.
当前不建议合并。以下是对已有意见的独立复核,不把重复问题计作新发现。
-
[P2] 无 PyYAML 时的回退不能保证打开原来命中的 Skill。 已在当前 head、Python 3.11.15 禁用 site packages 的环境复现:
find 'cuda graph warmup' --kinds skill --limit 1 --json选中kernel-writing-guide,路径为methods/kernel-writing-guide/SKILL.md;而按plugins/inferrsi/skills/inferrsi/SKILL.md:129-133和docs/operations.md:246-250执行collect --relevance 'cuda graph warmup' --limit 1 --material --json,返回的是inferrsi.ascend.skills.ascend_aclgraph_capture_replay。两个命令都成功退出,但材料已经换了。请通过选中 hit 的身份/路径确定对应 package projection,并校验skill_documents.original;或明确把回退描述为重新搜索、要求重新判断结果,不能称作读回该命中项。建议用这个例子补回归覆盖。确认 David 的现有意见 仍然成立。 -
必选
knowledge-model-policy检查仍失败。 已读取当前 head 的最新 declarations 日志,明确报告Material AI author 1: model is unapproved or suspended;也重新读取了 main 分支规则,确认该 context 仍是必选项。请保留真实的 unknown 声明,按仓库流程取得有权批准者的 exception,或提供符合规则的实际作者声明;普通代码 review 不能替代此项。 -
默认入口切换仍缺少记录下来的范围决定。 已读取 #379 正文及全部四条评论;最新提议 仍是在请求请求者/维护者接受
find作为默认入口,未见接受记录。这不表示 #380 对旧短指引的负结果证明新工具无效;问题是本 PR 将 README、quickstart 和插件默认指引切到find → read,而这项产品决定尚未记录。请在 #379 确认范围,或先保留原默认入口、把新命令作为增量能力交付。 -
当前分支存在合并冲突。 GitHub 返回
mergeable=false / mergeable_state=dirty;本地git merge-tree --write-tree HEAD origin/main在上述 fetched main 上也退出 1。当前实际冲突文件是src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json。请结合最新 main 重新生成一致的索引/provenance,处理冲突后在新 head 上重跑检查并请求复审。此处的冲突路径是本次复核结果,不沿用较早评审列出的路径。
验证:
- Python 3.11.15 / macOS,独立 detached worktree。运行 knowledge find/CLI/operations、skill catalog/index、Codex/WorkSwarm/DSH 插件及 module boundaries 共九个测试文件:217 passed, 2 skipped, 1 failed。
- 唯一失败为
tests/test_dsh_plugin.py::test_remote_ssh_connection_and_scoped_operations,其 Node 检查返回AssertionError ERR_ASSERTION。在单独的 base worktree 运行相同tests/check_dsh_remote.cjs也失败。因此不归因为本 PR 新增回归,但本次套件不是全绿。 ruff check src tests、git diff --check通过;tools/build_skill_index.py --source-revision 1431037fe138679a0b225e1501dbbc43f97a2150 --check通过(80 metadata records)。- 本地 editable 安装的 SCM 自动版本推导受已有非版本 Git 标签影响;仅测试安装使用
SETUPTOOLS_SCM_PRETEND_VERSION=0.0.0,没有修改受审代码。上述结果不作为默认打包成功的证明。 - 核对了新 findings 与仓库内原 Case 正文、检索评分、共享 reader/path 校验、精简读回和插件启动链;除上述已复现问题,未发现足够明确的新实现阻塞项。
- 未运行全仓测试、Windows 实机、GPU/NPU 或真实 Agent 对照;未独立验证 PR 正文的优化收益/成本数据。本次是 PR 工程审查,不是 #379 的目标验收或 campaign PASS。
Move the find integration to the current Apps CLI and clarify that the no-PyYAML fallback searches packaged skills independently. Refresh the generated provenance in the next commit from this combined source revision. Agent-assisted-by: Codex (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
…ression Agent-assisted-by: Codex (author) Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
关联需求与交付范围
Refs #379
Agent 描述优化中遇到的问题后,可以在一次查询里找到适用的指南、资料记录和历史案例,再用结果给出的命令读到带出处、适用条件、限制和版本的原文。本 PR 交付这个本地检索与读回流程。
原有搜索与观察到的问题
knowledge search/knowledge researchcollect --relevancecollect/read在已有回放里,相关 Case 曾在选择阶段落选。另一些已选材料在返回外层工具时出现过截断。对 Agent 来说,命中和拿到正文是两步。
本 PR 的改法
inferrsi find "<问题描述>"用同一套词项排序比较已登记的 Case、Skill 和 package Record。字段权重优先考虑标题、摘要、标签和 findings,预先计算词频并降低长正文的权重。查询支持词边界、常见分隔符、中英文别名和--kinds。findings,让检索可以直接命中尝试过的方向、结果、原因和复试条件。每条预览带记录中的适用范围或边界。open,JSON 提供open_argv。工具用同一 reader 检查返回项能否读到材料。详细的正文、来源和 revision 由read --material --omit-resources返回,需要清单时可以完整读取。find → read介绍常规发现流程。旧的精确收集、读取和资源接口继续服务现有调用方。Codex 和 WorkSwarm 插件沿各自的安装入口读回。例如:
结果会提示 Qwen Ascend 案例及其读取命令。读回正文时可取得 revision,供后续固定版本引用。
实测结果
下表在合并
main后的8c0a7bfc上复测。类型标注只统计预期有材料的查询。8c0a7bfc冻结查询包括 Qwen(5 条)和共享资源哈希(4 条)。返回正文和冻结原文逐字节一致,来源、适用条件和限制均保留。按返回的 revision 重读,结果一致。
用同一版本对照
package:inferrsi.common.skills.optimization_search_policy的完整和精简读回,JSON 从 223,782 字节降到 23,757 字节。正文、来源、适用条件和 revision 一致。Qwen-Image 单次真实 Agent 检索对照
基于 #338 的缩比
qwen21-denoise-npu题包,使用两个全新 GPT-6 Luna / max 会话,在同一张 Ascend910B2C 上串行比较旧session knowledge与本 PR 的find → read;两组都开启知识库,各完成 8 轮和独立 final。A 的上下文预算为 16,384 字节,B 从预览中选择并读回正文。每次评测均覆盖五个正确性 case、其中三个计时,pairs=2, samples=3, warmups=1。find → read本次新检索会话实际读取并采用了更多针对性材料:第 1 轮引用
vllm_omni.ascend.diffusion.qwen_image_21_eager_t2i(revisionsha256:b1740d8dd28f3c2f7ec2e4fbb9233a7ce311f52210fa3c6365bb320ad191e142),优先测试请求内 conditioning 复用;第 2–4 轮引用case.qwen-ascend-rejected-directions(revisionsha256:039a074c881a924327549416d143fcf7cb550493e87352113b45a08d8717827a),拆开验证 RoPE 和 mask 缓存。A 送达的package:inferrsi.ascend.skills.ascend_optimization_casebook(revisionsha256:7208ca2225432e313e1f899bde5b963f39d617b45996cd336da67238f8a5fa86)没有被记录为改变候选或验证决定。采用数按唯一材料计算;A 的另外四项只交付了命中元数据。**范围:**每组仅一次自主查询,提问、语料和交付方式不同,也没有独立相关性标注;这张表不能单独证明新检索整体更好或收益稳定。Campaign 耗时包含优化和评测,不是单次检索耗时;缓存输入包含在输入 token 中。表中 PASS 仅指题包 final 质量检查,不代表整个检索对照或真实图像生成、服务性能验收通过。正式 A 前另有一次中止的 Sol/high 会话,其三轮结果未纳入比较;两组在不同时间运行。原始证据保存在私有环境,尚未公开完整 evidence tree。
检查与结果边界
cuda graph warmup。find命中kernel-writing-guide,collect --relevance重新搜索打包语料,命中inferrsi.ascend.skills.ascend_aclgraph_capture_replay。指引要求先判断新结果是否适用,再用它的ref、skill_documents.original和revision经resource读回正文。两条评审查询的读回均与 wheel 原文逐字节一致,基础安装里 Case 的find → read也通过。 David 的评审main的891aadda上出现同样的失败。Ruff、diff 检查和生成索引校验通过。find → read作为默认查询和读取流程,旧命令用于回退。默认入口切换的决定待在 tracking: Skill Indexer 检索效率、准确性与逐轮采用 #379 记录。单次 feat: share RSI optimization management across project and benchmark tasks #338 对照观察到材料采用,稳定的检索改善和优化收益仍待验证。 #379 范围提议知识作者声明